{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image Input/Output with OpenAI V2 Client and AG2 Agents\n",
    "\n",
    "**Author:** Yixuan Zhai\n",
    "\n",
    "This notebook demonstrates how to use the new **OpenAI V2 Client** (`api_type: \"openai_v2\"`) with AG2's agent system for image input and vision tasks.\n",
    "\n",
    "## What is OpenAI V2 Client?\n",
    "\n",
    "The V2 client is a next-generation LLM client that implements both `ModelClient` and `ModelClientV2` protocols, returning rich `UnifiedResponse` objects with:\n",
    "\n",
    "- **Typed content blocks**: `TextContent`, `ReasoningContent`, `ToolCallContent`, `CitationContent`\n",
    "- **Forward compatibility**: `GenericContent` handles unknown future content types\n",
    "- **Rich metadata**: Full reasoning blocks, citations, and tool execution details\n",
    "- **Type safety**: Pydantic validation for all response data\n",
    "- **Cost tracking**: Automatic per-response cost calculation\n",
    "\n",
    "## Comparison with Standard OpenAI Client\n",
    "\n",
    "| Feature | Standard Client | V2 Client |\n",
    "|---------|----------------|----------|\n",
    "| Response Type | `ChatCompletion` (dict) | `UnifiedResponse` (Pydantic) |\n",
    "| Content Blocks | Untyped | Strongly typed |\n",
    "| Reasoning Support | String | `ReasoningContent` with metadata |\n",
    "| Agent Integration | ✅ Yes | ✅ Yes (via duck typing) |\n",
    "| Type Safety | Minimal | Full Pydantic validation |\n",
    "| Extensibility | Fixed | Forward-compatible |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "```bash\n",
    "pip install ag2[openai]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup: Create Assistant with V2 Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import textwrap\n",
    "\n",
    "from autogen import AssistantAgent, UserProxyAgent\n",
    "from autogen.io.run_response import Cost\n",
    "\n",
    "\n",
    "# Helper function to extract total cost from ChatResult.cost dict\n",
    "def get_total_cost(cost_dict):\n",
    "    \"\"\"Extract total cost from ChatResult.cost dict structure.\"\"\"\n",
    "    total = 0.0\n",
    "    for usage_type in cost_dict.values():\n",
    "        if isinstance(usage_type, dict):\n",
    "            for model_usage in usage_type.values():\n",
    "                if isinstance(model_usage, dict) and \"cost\" in model_usage:\n",
    "                    total += model_usage[\"cost\"]\n",
    "    return total\n",
    "\n",
    "\n",
    "# Helper function to extract cost from run response\n",
    "def get_total_cost_from_run(run_response_cost):\n",
    "    \"\"\"Extract total cost from run response object.\"\"\"\n",
    "    if isinstance(run_response_cost, Cost):\n",
    "        return run_response_cost.usage_including_cached_inference.total_cost\n",
    "    return 0.0\n",
    "\n",
    "\n",
    "# Configure LLM to use V2 client\n",
    "llm_config = {\n",
    "    \"config_list\": [\n",
    "        {\n",
    "            \"api_type\": \"openai_v2\",  # <-- Key: use V2 client architecture\n",
    "            \"model\": \"gpt-4o-mini\",  # Vision-capable model\n",
    "            \"api_key\": os.getenv(\"OPENAI_API_KEY\"),\n",
    "        }\n",
    "    ],\n",
    "    \"temperature\": 0.3,\n",
    "}\n",
    "\n",
    "# Create vision assistant\n",
    "assistant = AssistantAgent(\n",
    "    name=\"VisionBot\",\n",
    "    llm_config=llm_config,\n",
    "    system_message=textwrap.dedent(\"\"\"\n",
    "        You are an AI assistant with vision capabilities.\n",
    "        You can analyze images and provide detailed, accurate descriptions.\n",
    "    \"\"\").strip(),\n",
    ")\n",
    "\n",
    "# Create user proxy\n",
    "user_proxy = UserProxyAgent(\n",
    "    name=\"User\",\n",
    "    human_input_mode=\"NEVER\",\n",
    "    max_consecutive_auto_reply=0,\n",
    "    code_execution_config=False,\n",
    ")\n",
    "\n",
    "# Test image URL\n",
    "IMAGE_URL = \"https://upload.wikimedia.org/wikipedia/commons/3/3b/BlkStdSchnauzer2.jpg\"\n",
    "\n",
    "print(\"✓ Assistant with V2 client created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: Simple Image Description\n",
    "\n",
    "Using formal image input format to reduce hallucination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Formal image input format (recommended)\n",
    "message_with_image = {\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [\n",
    "        {\"type\": \"text\", \"text\": \"Describe this image in one sentence.\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}},\n",
    "    ],\n",
    "}\n",
    "\n",
    "# Initiate chat with image\n",
    "chat_result = user_proxy.initiate_chat(assistant, message=message_with_image, max_turns=1, summary_method=\"last_msg\")\n",
    "\n",
    "print(\"\\n=== Response ===\")\n",
    "print(chat_result.summary)\n",
    "print(f\"\\nCost: ${get_total_cost(chat_result.cost):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2: Detailed Image Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "detailed_message = {\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [\n",
    "        {\"type\": \"text\", \"text\": \"Analyze this image in detail. What breed is this dog? What are its characteristics?\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}},\n",
    "    ],\n",
    "}\n",
    "\n",
    "chat_result = user_proxy.initiate_chat(\n",
    "    assistant,\n",
    "    message=detailed_message,\n",
    "    max_turns=1,\n",
    "    clear_history=True,  # Start fresh conversation\n",
    ")\n",
    "\n",
    "print(chat_result.summary)\n",
    "print(f\"\\nCost: ${get_total_cost(chat_result.cost):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3: Multi-Turn Conversation with Image Context\n",
    "\n",
    "The assistant maintains context across turns - no need to resend the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First turn: Show image and ask initial question\n",
    "initial_message = {\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [\n",
    "        {\"type\": \"text\", \"text\": \"What animal is in this image?\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}},\n",
    "    ],\n",
    "}\n",
    "\n",
    "chat_result = user_proxy.initiate_chat(assistant, message=initial_message, max_turns=1, clear_history=True)\n",
    "\n",
    "print(\"Turn 1:\")\n",
    "print(chat_result.summary)\n",
    "\n",
    "# Second turn: Follow-up question (context maintained)\n",
    "followup = user_proxy.send(\n",
    "    message=\"What are the distinctive characteristics of this breed?\", recipient=assistant, request_reply=True\n",
    ")\n",
    "\n",
    "print(\"\\nTurn 2:\")\n",
    "print(followup)\n",
    "\n",
    "# Third turn: Another follow-up\n",
    "followup2 = user_proxy.send(message=\"Is this dog well-suited for families?\", recipient=assistant, request_reply=True)\n",
    "\n",
    "print(\"\\nTurn 3:\")\n",
    "print(followup2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 4: Using run() Interface\n",
    "\n",
    "The V2 client also works with the `run()` interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use run() interface\n",
    "response = assistant.run(\n",
    "    message={\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"text\", \"text\": \"What is the primary color of this dog's coat?\"},\n",
    "            {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}},\n",
    "        ],\n",
    "    },\n",
    "    user_input=True,\n",
    "    max_turns=1,\n",
    "    clear_history=True,\n",
    ")\n",
    "\n",
    "# Process the response\n",
    "response.process()\n",
    "\n",
    "print(\"=== Run Interface Result ===\")\n",
    "print(f\"Summary: {response.summary}\")\n",
    "print(f\"Cost: ${get_total_cost_from_run(response.cost):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 5: Multiple Images Comparison\n",
    "\n",
    "The V2 client can handle multiple images in a single request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Two different dog images\n",
    "IMAGE_URL_1 = \"https://upload.wikimedia.org/wikipedia/commons/3/3b/BlkStdSchnauzer2.jpg\"\n",
    "IMAGE_URL_2 = \"https://upload.wikimedia.org/wikipedia/commons/6/6e/Golde33443.jpg\"\n",
    "\n",
    "comparison_message = {\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [\n",
    "        {\"type\": \"text\", \"text\": \"Compare these two dogs. What are the differences between them?\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL_1}},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL_2}},\n",
    "    ],\n",
    "}\n",
    "\n",
    "chat_result = user_proxy.initiate_chat(assistant, message=comparison_message, max_turns=1, clear_history=True)\n",
    "\n",
    "print(chat_result.summary)\n",
    "print(f\"\\nCost for 2 images: ${get_total_cost(chat_result.cost):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 6: Group Chat with V2 Client\n",
    "\n",
    "The V2 client works seamlessly in multi-agent group chat scenarios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogen import ConversableAgent\n",
    "from autogen.agentchat.groupchat import GroupChat, GroupChatManager\n",
    "\n",
    "# Create specialized agents with V2 client\n",
    "image_analyst = ConversableAgent(\n",
    "    name=\"ImageAnalyst\",\n",
    "    llm_config=llm_config,\n",
    "    human_input_mode=\"NEVER\",\n",
    "    system_message=\"You analyze images. Keep responses brief and focused on visual details.\",\n",
    ")\n",
    "\n",
    "breed_expert = ConversableAgent(\n",
    "    name=\"BreedExpert\",\n",
    "    llm_config=llm_config,\n",
    "    human_input_mode=\"NEVER\",\n",
    "    system_message=\"You are a dog breed expert. Provide breed-specific information concisely.\",\n",
    ")\n",
    "\n",
    "# Create user proxy for group chat\n",
    "group_user = UserProxyAgent(\n",
    "    name=\"Coordinator\", human_input_mode=\"NEVER\", max_consecutive_auto_reply=0, code_execution_config=False\n",
    ")\n",
    "\n",
    "# Create group chat\n",
    "groupchat = GroupChat(\n",
    "    agents=[group_user, image_analyst, breed_expert], messages=[], max_round=4, speaker_selection_method=\"round_robin\"\n",
    ")\n",
    "\n",
    "# Create manager with V2 client\n",
    "manager = GroupChatManager(groupchat=groupchat, llm_config=llm_config)\n",
    "\n",
    "# Start group chat with image\n",
    "group_message = {\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [\n",
    "        {\"type\": \"text\", \"text\": \"Team, analyze this dog image and tell me about the breed.\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}},\n",
    "    ],\n",
    "}\n",
    "\n",
    "chat_result = group_user.initiate_chat(manager, message=group_message, max_turns=3)\n",
    "\n",
    "print(\"\\n=== Group Chat Result ===\")\n",
    "print(f\"Summary: {chat_result.summary}\")\n",
    "print(f\"Cost: ${get_total_cost(chat_result.cost):.4f}\")\n",
    "print(\"\\n=== Participants ===\")\n",
    "participant_names = {msg.get(\"name\") for msg in chat_result.chat_history if msg.get(\"name\")}\n",
    "print(f\"Agents participated: {participant_names}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 6b: Pattern-Based Group Chat\n",
    "\n",
    "The V2 client works seamlessly with AG2's modern pattern-based group orchestration API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 6b: Pattern-Based Group Chat (Modern API)\n",
    "\n",
    "AG2's pattern-based group chat API provides a modern, flexible way to orchestrate multi-agent conversations. The V2 client works seamlessly with this pattern system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogen.agentchat.group.multi_agent_chat import initiate_group_chat\n",
    "from autogen.agentchat.group.patterns import DefaultPattern\n",
    "\n",
    "# Create specialized agents with V2 client\n",
    "data_analyst = ConversableAgent(\n",
    "    name=\"DataAnalyst\",\n",
    "    llm_config=llm_config,\n",
    "    human_input_mode=\"NEVER\",\n",
    "    system_message=\"You analyze data and provide insights. Be brief and focused.\",\n",
    ")\n",
    "\n",
    "quality_reviewer = ConversableAgent(\n",
    "    name=\"QualityReviewer\",\n",
    "    llm_config=llm_config,\n",
    "    human_input_mode=\"NEVER\",\n",
    "    system_message=\"You review analysis quality and provide feedback. Be concise.\",\n",
    ")\n",
    "\n",
    "# Create pattern-based group chat\n",
    "pattern = DefaultPattern(\n",
    "    initial_agent=data_analyst,  # Starting agent\n",
    "    agents=[data_analyst, quality_reviewer],  # All agents in group\n",
    ")\n",
    "\n",
    "# Initiate group chat using pattern API\n",
    "chat_result, context_variables, last_agent = initiate_group_chat(\n",
    "    pattern=pattern,\n",
    "    messages=\"Analyze the number 42 and discuss its significance.\",\n",
    "    max_rounds=3,\n",
    ")\n",
    "\n",
    "print(\"\\n=== Pattern-Based Group Chat Result ===\")\n",
    "print(f\"Summary: {chat_result.summary}\")\n",
    "print(f\"Cost: ${get_total_cost(chat_result.cost):.4f}\")\n",
    "print(f\"Last speaker: {last_agent.name}\")\n",
    "print(\"\\n=== Participants ===\")\n",
    "participant_names = {msg.get(\"name\") for msg in chat_result.chat_history if msg.get(\"name\")}\n",
    "print(f\"Agents: {participant_names}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_cost = 0\n",
    "conversations = [\n",
    "    \"What animal is this?\",\n",
    "    \"What breed specifically?\",\n",
    "    \"What color is it?\",\n",
    "]\n",
    "\n",
    "# First, show the image\n",
    "chat_result = user_proxy.initiate_chat(\n",
    "    assistant,\n",
    "    message={\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [{\"type\": \"text\", \"text\": conversations[0]}, {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}}],\n",
    "    },\n",
    "    max_turns=1,\n",
    "    clear_history=True,\n",
    ")\n",
    "total_cost += get_total_cost(chat_result.cost)\n",
    "print(f\"Q1: {conversations[0]}\")\n",
    "print(f\"A1: {chat_result.summary}\")\n",
    "print(f\"Cost: ${get_total_cost(chat_result.cost):.4f}\\n\")\n",
    "\n",
    "# Follow-up questions (image context maintained)\n",
    "for question in conversations[1:]:\n",
    "    response = user_proxy.send(message=question, recipient=assistant, request_reply=True)\n",
    "    print(f\"Q: {question}\")\n",
    "    print(f\"A: {response}\\n\")\n",
    "\n",
    "print(f\"=== Total Conversation Cost: ${total_cost:.4f} ===\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 8: Accessing the UnifiedResponse (Advanced)\n",
    "\n",
    "While agents work with the V2 client transparently, you can also use the client directly for advanced features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the V2 client directly\n",
    "from autogen.llm_clients import OpenAICompletionsClient\n",
    "\n",
    "# Create client directly\n",
    "v2_client = OpenAICompletionsClient(api_key=os.getenv(\"OPENAI_API_KEY\"))\n",
    "\n",
    "# Make a direct call\n",
    "response = v2_client.create({\n",
    "    \"model\": \"gpt-4o-mini\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": [\n",
    "                {\"type\": \"text\", \"text\": \"Describe this image.\"},\n",
    "                {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}},\n",
    "            ],\n",
    "        }\n",
    "    ],\n",
    "})\n",
    "\n",
    "print(\"=== Direct V2 Client Access ===\")\n",
    "print(f\"Response Type: {type(response).__name__}\")\n",
    "print(f\"Response ID: {response.id}\")\n",
    "print(f\"Model: {response.model}\")\n",
    "print(f\"Provider: {response.provider}\")\n",
    "print(f\"\\nText: {response.text}\")\n",
    "print(\"\\n=== Typed Content Blocks ===\")\n",
    "print(f\"Number of messages: {len(response.messages)}\")\n",
    "print(f\"Message role: {response.messages[0].role}\")\n",
    "print(f\"Content blocks: {len(response.messages[0].content)}\")\n",
    "for i, block in enumerate(response.messages[0].content):\n",
    "    print(f\"  Block {i}: {block.type} - {type(block).__name__}\")\n",
    "\n",
    "print(\"\\n=== Rich Metadata ===\")\n",
    "print(f\"Reasoning blocks: {len(response.reasoning)}\")\n",
    "if hasattr(response, \"tool_calls\"):\n",
    "    print(f\"Tool calls: {len(response.tool_calls)}\")\n",
    "print(f\"Usage: {response.usage}\")\n",
    "print(f\"Cost: ${response.cost if response.cost is not None else 0:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "### Key Benefits of V2 Client with Agents\n",
    "\n",
    "1. **Seamless Integration**: Works with existing `AssistantAgent` and `UserProxyAgent`\n",
    "2. **Rich Response Data**: Access to typed content blocks (reasoning, citations, etc.)\n",
    "3. **Vision Support**: Full multimodal capabilities with formal image input\n",
    "4. **Cost Tracking**: Automatic per-response cost calculation\n",
    "5. **Type Safety**: Pydantic validation for all response data\n",
    "6. **Forward Compatible**: GenericContent handles unknown future types\n",
    "7. **Duck Typing**: UnifiedResponse works with agent system via duck typing\n",
    "\n",
    "### Usage Pattern\n",
    "\n",
    "```python\n",
    "# Simple: Just change api_type\n",
    "llm_config = {\n",
    "    \"config_list\": [{\n",
    "        \"api_type\": \"openai_v2\",  # <-- That's it!\n",
    "        \"model\": \"gpt-4o-mini\",\n",
    "        \"api_key\": \"...\",\n",
    "    }]\n",
    "}\n",
    "\n",
    "assistant = AssistantAgent(llm_config=llm_config)\n",
    "# Everything works as before, but with rich UnifiedResponse internally\n",
    "```\n",
    "\n",
    "### When to Use V2 Client\n",
    "\n",
    "- ✅ Need access to reasoning blocks (o1/o3 models)\n",
    "- ✅ Want typed, structured response data\n",
    "- ✅ Building systems that need forward compatibility\n",
    "- ✅ Require rich metadata and citations\n",
    "- ✅ Working with vision/multimodal models\n",
    "\n",
    "### Migration from Standard Client\n",
    "\n",
    "No code changes needed! Just update `api_type`:\n",
    "\n",
    "```python\n",
    "# Before\n",
    "{\"model\": \"gpt-4o-mini\", \"api_key\": \"...\"}\n",
    "\n",
    "# After  \n",
    "{\"api_type\": \"openai_v2\", \"model\": \"gpt-4o-mini\", \"api_key\": \"...\"}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "front_matter": {
   "description": "OpenAI V2 Client",
   "tags": [
    "openai",
    "clients"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
